Research question 3: Role of set size 1

Experiment: AVA11

Leaving out individual set sizes

We will first look at the effect of leaving out individual set sizes on the in-sample fits and out-of-sample prediction of the data, and associated predictions of all models. Here, all models are fit to all (remaining) set sizes simultaneously, i.e., mean memory precision changes for individual set sizes is derived from mean memory precision for a single item. Please see the manuscript for a more detailed explanation of the approach shown here.

Experiment-level data

By experiment-level data, we mean the data in AVA11 as a collection of the data provided by all 45 participant(s). The manner in which data, model fitting results and predictions of an experiment are sumamrized can vary depending on goal and approaches. For some aspects of the data, some approaches make more sense than others. For completeness’ sake we show a variety here, even if some are somewhat non-sensical.

Aggregate Fits

Here we treated all data in AVA11 as if it had been provided by a single participant (rather than provided by 45 participant(s)). We fitted all models to this aggregate data. The model comparison shows relative model fits on the basis of the fit to the aggregate data.

In the error distribution and summary statistics, the data is given by the observed data aggregated across individuals. The predictions (error distributions, resultant summary statistics and normalized RMSD relative to the aggregate data) are made on the basis of aggregate fit best-fit parameter estimates for all models.

Averaging individual parameter estimates

In this tab, all graphs are shown for the purpose of illustrating the performance in the experiment on average, i.e., averaged across participants.

The model comparison plot is based on determining and averaginfg the relative difference of all individual model fits to the best-fit model. Across the data corpus some participants produced extreme out-of-sample predictions (> 10000 points) for some held-out set sizes. Where this is the case we present two model comparison graphs. (A) shows the model comparison with all participants included, and where applicable (B) shows the model comparison for those participants excluded for all panels.

Here in AVA11, we excluded 4 participant(s) (AVA11_E4_19, AVA11_E4_21, AVA11_E4_23, AVA11_E4_17) for extreme out-of-sample predictions (> 10000 points); the number of participants in each panel is shown at the top of each panel.

The predictions of the behavioral signature pattern (error distribution and resultant summary statistics) are based on averaging the parameter estimates of the best-fit individual fits. The observed data is given by the aggregate data for both graphs. If any participants were excluded for extreme out-of-sample prediction in the model comparison, they were also excluded here. Thus the graphs below represent the data and predictions of 41 participants.

Averaging individual predictions

In the tab “Averaging individual parameter estimates”, we looked at the prediction of error distributions and resultant summary statistics on the basis of averaging participants’ best-fit parameter estimates. Here, the predictions are based on averaging individual’s predictions (which were based on best-fit parameter estimates). We excluded 4 participant(s) (AVA11_E4_19, AVA11_E4_21, AVA11_E4_23, AVA11_E4_17) as in “Averaging individual parameter estimates” before aggregating/averaging individuals’ predictions. For the summary statistics (across set sizes, and normalized RMSD) we show two graphs: A) deriving the summary statistics from the averaged error distributions (for NRMSDs compared to the summary statistics of the aggregate data), and B) averaging individuals’ summary statistics, and similarly aggregating individuals’ normalized RMSDs.

Plots labeled “A”. The following summary statistics are derived from the averaged error distribution above. The graphs with the normalized NRMSD shows the comparison of these averaged prediction to the aggregate data. This is not particularly useful as the distribution underlying the aggregate data is not necessarily the averaged of the individuals’ data, but it is one approach to summarizing the data.

Plots labeled “B”. The following summary statistics are based on averaging participants’ summary statistics, for both the data (e.g., the summary statistics derived from individual observed error distributions) and the model predictions (e.g., the summary statistics derived from individual observed error distributions). The graphs with the normalized NRMSD shows the normalized RMSD across individuals as boxplots to provide an idea of the spread of the NRMSD in the experiment for these models (i.e., NOT the normalized RMSD derived from contrasting averaged predicted and observed summary statistics.)

Individual-level data

AVA11_E4_1

AVA11_E4_10

AVA11_E4_11

AVA11_E4_12

AVA11_E4_13

AVA11_E4_14

AVA11_E4_15

AVA11_E4_16

AVA11_E4_17

AVA11_E4_18

AVA11_E4_19

AVA11_E4_2

AVA11_E4_20

AVA11_E4_21

AVA11_E4_22

AVA11_E4_23

AVA11_E4_24

AVA11_E4_25

AVA11_E4_26

AVA11_E4_27

AVA11_E4_28

AVA11_E4_29

AVA11_E4_3

AVA11_E4_30

AVA11_E4_31

AVA11_E4_32

AVA11_E4_33

AVA11_E4_34

AVA11_E4_35

AVA11_E4_36

AVA11_E4_37

AVA11_E4_38

AVA11_E4_39

AVA11_E4_4

AVA11_E4_40

AVA11_E4_41

AVA11_E4_42

AVA11_E4_43

AVA11_E4_44

AVA11_E4_45

AVA11_E4_5

AVA11_E4_6

AVA11_E4_7

AVA11_E4_8

AVA11_E4_9

Fitting set sizes separately

It is possible that the effect of set size 1 is due to the data at set size 1 being particularly diagnostic of a difference between models. We therefore fitted the VP models with unlimited memory capacity (+/- models) to the data of all set sizes separately. This means that all parameters were estimated separately for all set sizes, and mean memory precision was not assumed to be linked across set sizes by the power function.

We compared models’ fit in deviance terms, with the relative difference between models when data from all set sizes was fit simultaneously.

Experiment-level data

Aggregate Fits

Averaging individual paramater estimates

Averaging individual predictions

A. Based on the summary statistic derived from averaged error distributions.

B. Based on summary statistics averaged across participants (derived from individual best-fit predictions of the error distribution)

Individual-level data

AVA11_E4_1

AVA11_E4_10

AVA11_E4_11

AVA11_E4_12

AVA11_E4_13

AVA11_E4_14

AVA11_E4_15

AVA11_E4_16

AVA11_E4_17

AVA11_E4_18

AVA11_E4_19

AVA11_E4_2

AVA11_E4_20

AVA11_E4_21

AVA11_E4_22

AVA11_E4_23

AVA11_E4_24

AVA11_E4_25

AVA11_E4_26

AVA11_E4_27

AVA11_E4_28

AVA11_E4_29

AVA11_E4_3

AVA11_E4_30

AVA11_E4_31

AVA11_E4_32

AVA11_E4_33

AVA11_E4_34

AVA11_E4_35

AVA11_E4_36

AVA11_E4_37

AVA11_E4_38

AVA11_E4_39

AVA11_E4_4

AVA11_E4_40

AVA11_E4_41

AVA11_E4_42

AVA11_E4_43

AVA11_E4_44

AVA11_E4_45

AVA11_E4_5

AVA11_E4_6

AVA11_E4_7

AVA11_E4_8

AVA11_E4_9